Frequency, Contingency and the Information Processing Theory of Conditioning

The framework provided by Claude Shannon’s (1948) theory of information leads to a far-reaching, more quantitatively oriented reconceptualization of the processes that mediate what is commonly called associative learning. The focus shifts from processes set in motion by individual events to processes sensitive to the information carried by the flow of events. The conception of what properties of the conditioned and unconditioned stimuli are important shifts from the tangible properties that excite sensory receptors to the abstract and intangible properties of number, duration, frequency and contingency, which are the carriers of the information. Frequency is an abstraction built on abstractions—one intangible, number, divided by another intangible, time. A contingent frequency raises the pyramid of abstractions still higher. It is the rate at which an event occurs following the onset or offset of a conditioning event. Sensitivity to contingent frequency requires frequency estimation together with the estimation of contingency, which is itself a forbidding abstraction (defined below). Nonetheless, it has begun to appear that brains routinely compute conditional (contingent) frequencies. Their ability to do so may explain much that has been seen as the work of associative formation, a process that does not operate on abstractions like number, time and contingency. In 1968, Robert Rescorla (1968) reported a simple experiment that changed in fundamental ways our conception of what has generally been called the associative process, the process that mediates Pavlovian conditioning, and, arguably, much else. The implications were so unsettling that they have not to this day been well digested by the community that studies learning, particularly by those that study it from a neurobiological perspective. What the results suggested was that the simple learning observed in Pavlovian conditioning paradigms arises from an information processing system sensitive to the contingencies between stimuli. If this implication is valid, then it changes our conception of the level of abstraction at which this basic learning process operates. Rescorla’s experiment followed from his reflections on the proper control procedures in Pavlovian conditioning experiments, published the previous year (Rescorla, 1967). Excitatory Pavlovian conditioning pairs a motivationally neutral stimulus (CS), like a tone or the illumination of a key, with a motivationally important stimulus (US), Information Processing in Conditioning 2 like a mildly painful shock to the feet or the delivery of food. The tone comes on and some while later the shock is felt; or, the key is illuminated and some while later food is delivered. After a few or several such experiences, the rat freezes and defecates when the tone comes on, and the pigeon delivers food pecks on the illuminated key, suggesting in both cases that the subject anticipates the imminent appearance of the US. The learning in these paradigms appears to be a simple manifestation of the basic associative conception, which is that new conducting links are created in the nervous system by the temporal pairing of stimulus events. By these links, the CS is excites the conditioned response. Therefore, Pavlovian conditioning has long been the paradigm of choice for investigating the properties of association formation. That temporally pairing the CS and US was a sine qua non for association formation was long assumed to be self-evident (and it still is in some neurobiological circles--cf. Usherwood, 1993). To prove that a change in behavior was a result of association formation, experimenters typically ran an experimental condition in which two stimuli were repeatedly presented together (temporally paired) and a control condition in which they were widely separated in time (not temporally paired). If the change was seen in the experimental condition but not in the control condition, then the underlying process was said to be associative. Temporal pairing and contingency had been tacitly assumed to be one and the same thing, until Rescorla (1967) pointed out that they could be dissociated, but that the commonly used control procedures did not do so. Control conditions in which the CS and US are never paired do not eliminate CS-US contingency, they replace one contingency with another. Rescorla (1967) pointed out that to determine whether it is temporal pairing or contingency that drives the conditioning process, one has to use the truly random control In this control, the occurrence of the CS does not restrict in any way the time at which the US can occur, so the US must sometime occur together with the CS, assuming that the CS is not a point event. Thus, the truly random control eliminates contingency but not temporal pairing, whereas the usual control eliminates temporal pairing but not contingency. Rescorla (1968) realized that the most interesting version of the truly random control would be one that did not affect at all the temporal pairing in the usual experimental condition. He achieved this as follows: The experimental condition (really, now, the control) was approximately the usual condition for producing what is called the conditioned emotional reaction (more simply, conditioned fear). Hungry rats were first trained to press a lever in an experimental chamber to obtain food, until they did so readily and steadily throughout two-hour sessions. Then came five sessions during which the lever was blocked. In each of these sessions, twelve 2-minute long tones came on at more or less random intervals during each session. During these sessions, subjects also experienced occasional very short, mildly painful shocks to their feet. What Rescorla manipulated was the distribution of the shocks relative to the tones. For one group, the shocks, 12 of them per session, were completely contingent on the tone, they only occurred when it was on. The rats in a second group, a truly random group, also got 12 shocks each session while the tone was on, and they also got shocks at an equal frequency (0.5/minute) during the intervals when the tone was not on. This protocol did Information Processing in Conditioning 3 not alter the number or frequency of tone-shock pairings, but it eliminated the tone-shock contingency. It also greatly increased the total number of shocks per session. To check whether that mattered, Rescorla ran a second truly random group: They got 12 shocks (the same total as the first group) but distributed at random without regard to the tone. Before testing for the extent to which the rats in the different groups had learned to fear the tone, Rescorla first eliminated their fear of the experimental chamber, with two more sessions in which the lever was unblocked and there were no tones and no shocks. By the end of these two sessions, the rats had resumed pressing the lever for food. In several final sessions, the rats’ conditioned fear of the tone was then measured by the effect of the tone on their willingness to continue pressing. If they feared the tone, they froze when it came on, and did not resume pressing until it went off. Although the rats in the first two conditions had the same tone-shock pairings, the rats in the contingent condition learned to fear the shock, whereas the rats in the truly random condition did not, nor did the rats in the other truly random condition. Thus, it is contingency and not temporal pairing that drives simple Pavlovian conditioning. What is commonly called inhibitory conditioning also implies that simple conditioned behavior is a consequence of contingency, not temporal pairing. It is called inhibitory conditioning because the protocols for producing it are inverses of excitatory conditioning protocols. They are such that when the CS occurs, the US will not occur. The simplest protocol is the explicitly unpaired protocol: the US occurs only when the CS is absent (e.g. Rescorla, 1966). The conditioned response—what the animal learns to do or not do-is the inverse of the excitatory response (LoLordo & Fairless, 1985; Rescorla, 1969).: The subject approaches and manipulates an inhibitory CS when it predicts a motivationally negative US. (For example, subjects approach the speaker from which the tone emanates if shocks never occur while the tone is on.) On the other hand, subjects avoid or refuse to manipulate a CS when it predicts that a positive US will not occur. (For example, if food never occurs when the key is illuminated, pigeons learn to distance themselves from the illuminated key.) Students of basic learning have always recognized that inhibitory conditioning was just as fundamental as excitatory conditioning (Hull, 1929; Pavlov, 1928), but they have not faced the full implications of the fact that it results from systematically not pairing the CS and the US. Between them, inhibitory conditioning experiments and the truly random control experiments demonstrate that temporal pairing is neither necessary nor sufficient to produce associative learning (for reviews of the evidence, see Gallistel, 1990; Gallistel & Gibbon, 2000). What does appear to be necessary and perhaps sufficient is contingency. Why has evidence that contingency not temporal pairing drives simple conditioning been so unsettling? Rescorla (1972, p. 10) put his finger on the problem, when he wrote, "We provide the animal with individual events, not correlations or information, and an adequate theory must detail how these events individually affect the animal. That is to say that we need a theory based on individual events." In other words, within the framework in which we have been wont to think about learning, the process had to operate at the level of events. It could not operate at the level of correlations and Information Processing in Conditioning 4 information. We do in fact provide the animal with correlations and information in conditioning experiments, because when we construct a conditioning protocol we correlate the events in such a way that the CS provides information about the temporal locus of the US. What Rescorla presumably meant was not that we do not provide the animals with information, but rather that we take it for granted that the learning processes does not operate at that level of abstraction; we assume that it sensitive only to the individual (physically definable?) events, not to the information they convey. In sum, the evidence that conditioning is driven by contingency unsettles us because it challenges what we have taken for granted about the level at which basic learning processes operate. Contingency, like number, arises at a more abstract level than the level of individual events. It can only be apprehended by a process that operates at the requisite level of abstraction--the level of information processing. A US is contingent on a CS to the extent that the CS provides information about the timing of the US. The information an event provides to a subject is measured by the reduction in the subject’s uncertainty (Shannon, 1948). This way of defining information leads directly to a rigorous quantification of information. It also accords with our every day intuition that the more information we have, the less uncertain we are. However, a disturbing feature of this definition is its subjectivity. The information conveyed by an event cannot be specified without reference to the subject’s prior knowledge. What that means in the present case is that if the CS does not reduce the subject’s uncertainty about the timing of the US, then it does not provide information. The conceptual upheaval in the study of learning in the late 1960s was the result of a series of experiments demonstrating that whether a conditioned response developed or not depended on the information about the US provided by the CS (L. J. Kamin, 1969; Rescorla, 1968; Wagner, Logan, Haberlandt, & Price, 1968). These experiments were unsettling because they suggested that the conditioning process manifests the subjectivity inherent in Shannon’s definition of information. Learning theorists have been struggling with these implications ever since (Barnet, Grahame, & Miller, 1993; Mackintosh, 1975; Pearce & Hall, 1980; Rescorla & Wagner, 1972; Wagner, 1981). What they have not done, however, is adopt the conceptual framework proposed by Shannon, despite its seeming relevance , its mathematical rigor, and its success in physics and engineering. The reluctance to adopt this framework appears to flow from two sources: dislike of the subjectivity inherent in it and a reluctance to believe that simple learning processes operate at so abstract a level. Here, I analyze some central issues in the study of conditioning from an information theoretic perspective. Understanding Conditioning from an Information Theoretic